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TECHNICAL REPORT 


The 

RICIS 

Concept 


The University of Houston-Clear Lake established the Research Institute for 
Computing and Information systems in 1986 to encourage NASA Johnson Space 
Center and local industry to actively support research in the computing and 
information sciences. As part of this endeavor, UH-Clear Lake proposed a 
partnership with JSC to jointly define and manage an integrated program of research 
in advanced data processing technology needed for JSCs main missions, including 
administrative, engineering and science responsibilities.^JSC agreed and entered i nJa 
a three-year cooperative agreement with UH-Clear Lake beginning in May, 1986, to 
jointly plan and execute such research through RICIS. Additionally, under 
Cooperative Agreement NCC 9-16, computing and educational facilities are shared 
by the two institutions to conduct the resear ch. 


The mission of RICIS is to conduct, coordinate and disseminate research on 
computing and information systems among researchers, sponsors and users from 
UH-Clear Lake, NASA/JSC, and other research organizations. Within UH-Clear 
Lake, the mission is being implemented through interdisciplinary involvement of 
faculty and students from each of the four schools: Business, Education, Human 
Sciences and Humanities, and Natural and Applied Sciences. 

Other research organizations are involved via the “gateway” concept. UH-Cl^r 
Lake establishes relationships with other universities and research organizations, 
having common research interests, to provide a33IfidhaTs6u fees of expSrtise^^ 


conduct needed research. 

A major role of RICIS is to find the best match of sponsors, res earchers and 
research objectives to advance knowledge in the computing and information 
sciences. Working jointly with NASA/JSC, RICIS advises on research needs, 
recommends principals for conducting the research, provides technical and 
administrative support to coordinate the research, and integrates technical results 
into the cooperative goals of U H-Clea r Lake and NASA/J$Cr~ - — 
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This research was conducted under auspices of the Research Institute for 
Computing and Information Systems by the International Business Machines 
Corporation. Dr. Terry Feagin and Dr. T. F. Leibfried served as RICIS research 
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Funding has been provided by Information Technology Division, 
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Clear Lake. The NASA technical monitor for this activity was Chris Culbert, of 
the Software Technology Branch, Information Technology Division, Information 
Technology Directorate, NASA/JSC. 

The views and conclusions contained in this report are those of the author 
and should not be interpreted as representative of the official policies, either 
express or implied, of NASA or the United States Government. 




Expert Systems Evaluation Questionnaire (Developer) 

By filling out this NASA funded questionnaire, you can help define the state-of-the- 
practice in the formal evaluation of Expert Systems on current NASA and industry 
applications. The information that you provide will be merged with the information 
from all other surveyed projects for the purpose of recommending future research 
and development activities. Individual responses are used solely as input to this 
information merging process. 

Expert System applications are becoming more prevalent in fields where proper 
functioning is essential, such as the medical, financial, and aerospace industries. It is 
widely claimed that Expert Systems are not as rigorously evaluated as traditional 
software because of unique, unresolved evaluation issues. To ensure the continued 
and safe deployment of Expert Systems into critical areas, adequate evaluation tech- 
niques which address these issues must be developed and performed. 

The answers to this questionnaire, together with follow-up interviews, will provide 
realistic answers to the following questions: 

• How much evaluation is being performed? 

• What evaluation techniques are in use? 

• What, if any, are the unique issues in evaluating Expert Systems? 

Instructions 

The following questions concern your experiences with an Expert System, either as 
a developer or as the manager of the development effort. Feel free to indicate your 
answers in any way you like. Some of the choices on the multiple choice questions 
have places to fill in additional information; please indicate the choice and include 
the additional information, if possible. 

This survey task must be performed within a relatively short time period. If pos- 
sible, please return completed questionnaires within one week of receipt to: 

Keith Kelley 
MC 6272A 

IBM Federal Solutions Division 
3700 Bay Area Blvd. 

Houston, Tx. 77058-1199 

If you have any questions regarding this questionnaire, please contact Keith at (713) 
282-7303. Each participating project may request a copy of the final survey report 
from Keith. 
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Questions 


1. What is the name of the Expert System you were/are involved with? 


2. Were you a developer of the Expert System or the manager of the develop- 
ment organization? 

a. Developer of Expert System b. Manager of Expert System 

development organization 

3, The responses that you provide in this questionnaire may indicate that further 
discussion is required for us to understand the issues that you encountered 
during the evaluation process. Would you be available, at your convenience, 
to discuss the evaluation process in more detail? Interviews will be fairly short 
one-on-one meetings either in person or by telephone. 

a. No, I am not available for an interview. 

b. Yes, I am available. 

Name 

Phone 


4. What field does the problem belong to? 


a. 

Aerospace 

g- 

Medical 

b. 

Financial 

h. 

Personnel 

c. 

Information Systems 

i. 

Research 

d. 

Hardware 

' j- 

Service 

e. 

Manufacturing 

k. 

Software 

f. 

Marketing 

1 . 

Other 


5. Which of the following items best describes what the Expert System does? 
Please indicate primary purpose with a and check all other applicable pur- 
poses (if any). 

a. Design - Configuring objects under constraints 

b. Repair - Executing plans to administer prescribed remedies 

c. Control - Governing overall system behavior 

d. Planning - Designing actions 

e. Diagnosis - Inferring system malfunctions from observables 

f. Debugging - Prescribing remedies for malfunctions 

g. Prediction - Inferring likely consequences of given situations 

h. Monitoring - Comparing observations to expected outcomes 

i. Instruction - Diagnosing, debugging, and repairing behavior 

j. Interpretation - Inferring situation descriptions from sensor 

k. Classification - Categorizing objects by properties data 


6. Some problems require the use of certainty factors (also called probabilities, or 
“fuzzy logic”) in their processing. Facts which contain certainty factors have 
the form: "if a is true, then there is an x% chance that b is true.” Does the 
Expert System include certainty factors? 

a. Yes c. I don't know 

b. No 

7. How much of the total problem space is the Expert System expected to 
address? I.e., if the Expert System is supposed to be able to diagnose 100 mal- 
functions, but the total number of malfunctions is known to be 200, the 
Expert System is expected to address 50% of the problem space. 



a. 

100% 

f. 

60% to 80% 


b. 

> 99% 

g- 

40% to 60% 

, — 

c. 

95% to 99% 

h. 

Other % 


d. 

90% to 95% 

i. 

I don't know 


e. 

80% to 90% 




8. What is your estimate of the percentage of the 

problem space that the Expert 


System actually covers? 



— 

a. 

Same as expected 

f. 

80% to 90% 


b. 

100% 

g- 

60% to 80% 


c. 

> 99% 

h,' 

40% to 60% 


d. 

95% to 99% 

i. 

Other % 


e. 

90% to 95% 

j- 

I don't know 


9. When developing an Expert System, the person w'ho provides the knowledge 
that is to be captured in the system is called the expert . For that part of the 
problem space addressed by the Expert System, how often is the expert(s) 
expected to give the correct answer? I.e., referring to question 7, how often 
does the expert(s) identify the correct malfunction out of the 100 addressed 
malfunctions? 



a. 

'Correct' defined by expert 

f. 

60% to 80% 


b. 

> 99% 

g- 

40% to 60% 


c. 

95% to 99% 

h. 

Other % 


d. 

90% to 95% 

i. 

I don't know 

— 

e. 

80% to 90% 



--- 

10. For that part of the problem space addressed by the Expert System, how often 


is the 

: Expert System expected to provide the correct answer? 



a. 

100% 

f. 

60% to 80% 


— 

b. 

> 99% 

g- 

40% to 60% 



c. 

95% to 99% 

h. 

Other 

% 


d. 

90% to 95% 

i. 

I don't know 


LL =" 

e. 

80% to 90% 
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1 1. What is your estimate of the percentage of the time that the Eepert System 
provides the correct answer for that part of the problem space addressed by the 
Expert System? 


a. 

100% 

f. 

60% to 80% 

b. 

> 99% 

g- 

40% to 60% 

c. 

95% to 99% 

h. 

Other 

d. 

90% to 95% 

i. 

I don't know 

e. 

80% to 90% 




12. What was the basis for determining how the system was to behave? 

a. A pre-existing document 

b. A requirements document completed as part of development. 

c. Some other developed document 

d. A prototype of the system 

e. Expert consultation 

f. Other 


13. Was there more than one expert consulted during the development of the 



system? 





a. 

System was developed by 

c. 

Multiple experts with lead 




expert 

d. 

Committee of experts 



b. 

Single expert 

e. 

Other 


14. 

How much interaction was there between the 

expert(s) and the development 



team? 



— 


a. 

Constant 

d. 

Occasional 



b. 

Frequent 

e. 

None 

..._ 


c. 

Regular 




15. 

Was the developers) part of the user organization? 




a. 

Yes 

c. 

User organization participated 



b. 

No 


in development 


16. 

Please indicate which development model was used for developing the Expert 



System. 





a. 

Traditional waterfall life-cycle 



= 


b. 


c. 


d. 


e. 


Requirements gathered before development of a prototype. A second 
requirements activity preceded Design, Implementation, and Test. 

Repetition of the Requirements, Design, Rule Generation, and Proto- 
typing phases until production system (final prototype) was developed. 

No effort was made to follow a particular model. 

Other 
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17. What percentage of the total development effort was dedicated to the each of 

the three parts of the Expert System? 

a. Information Structures % (Declarative part which represents 

the knowledge of the Expert System.) 

b. Inference Engine % (Processes the knowledge base to infer a 

set of output facts from a set of input facts. If an Expert System shell 
was used, this value should be 0%.) 

c. Traditional Code % (Used to supplement the inference 

process (e.g., interfacing the inference engine to a device or user, per- 
forming arithmetic calculations, etc.).) 

18. What was the primary language/tool for each part of the Expert System? 

a. Knowledge Base 

b. Inference Engine 

c. Traditional Code 

19. How hard was it to develop the original concept of what the system was sup- 
posed to do? 



a. 

Trivial 

d» Hard 

— 

b. 

Easy 

c. Impossible 


c. 

Medium 



20. Aside from any difficulties in developing the original concept, how hard was it 
to express the behavior (through the knowledge base)? 



a. 

Trivial 

d. Hard 


b. 

Easy 

e. Impossible 


c. 

Medium 



21. When changes were made to the knowledge, how often did some unexpected 
change occur? 


a. 

Never 

d. Usually 

b. 

Occasionally 

e. Always 

c. 

Frequently 



22. How were changes the the Expert System distributed to the users? 

a. Developers made changes to users' system. 

b. Tested system distributed to the users. 

c. Other 
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23. If the Expert System is rule-based, how many rules are contained in the 

Knowledge Base (KB)? 

If the Expert System is not rule-based, please give an approximate size of the 
knowledge base . 

The answer just given is: 

a. The actual effort c. A guess 

b. A very close approximation 

24. How much effort was expended in developing the system? (Please take a guess 

if you don't know) person/months. 

The indicated effort is: 

a. The actual effort c. 

b. A very close approximation 

25. Were any evaluation activities performed on 
developed? (indicate any that apply) 

a. No evaluation was performed e. 

b. Desk checking 

c. Formal inspections ^ 

d. Checked by expert(s) 

26. What evaluation activities were performed on the executing system after devel 
opment was completed? (indicate any that apply) 


a. 

No evaluation was performed d. 

User acceptance 

b. 

Checked by expert(s) e. 

Other 

c. 

Compared with documented 
behavior 


During evaluation, the results from executing the system were compared with: 

a. 

Requirements document d. 

Majority opinion of experts 

b. 

System prototype e. 

Other 

c. 

Single expert 


What was the level of agreement among the experts concerning the correctness 
of the system? That is, do the experts agree on the correctness of the results of 
the system? Please note that this does not mean that the experts agree with 
the system, but rather, that they agree with each other about the results of the 
system. 

a. 

Always agree c. 

A single expert was involved 

b. 

Agree % of the time. 



A guess 

the system while it was being 

Compared with documented 
behavior 

Structural testing (e.g. cover all 
rules) 

Other 
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29. How hard was the evaluation effort to perform? 


a. 

Trivial 

d. 

Hard 

b. 

Easy 

e. 

Impossible 

c. 

Medium 



How much effort was expended by the development organization in evaluating 
the correctness of the Expert System? (Please take a guess if you don't know) 


person/months. 



The indicated effort is: 



a. 

The actual effort 

c. 

A guess 

b. 

A very close approximation 



What is the worst thing that can 
answer? 

happen if the 

Expert System gives the wrong 

a. 

Someone gets hurt 

d. 

Work-around must be used 

b. 

Loss of ''mission* 

e. 

Nothing 

c. 

Nuisance (correct answer 

f. 

Can't tell the answer is wrong 


derived some other way) 

g- 

Other 


32. How does the number of errors that the users encounter compare with the 
number of errors they encounter with other systems which are not Expert 
Systems? 

a. Significantly more errors d. Fewer errors 

b. More errors e. Significantly fewer errors 

c. About the same number of f. I don't know 

errors 
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Expert Systems Evaluation Questionnaire (User) 

By filling out this NASA funded questionnaire, you can help define the state-of-the- 
practice in the formal evaluation of Expert Systems on current NASA and industry 
applications. The information that you provide will be merged with the information 
from all other surveyed projects for the purpose of recommending future research 
and development activities. Individual responses are used solely as input to this 
information merging process. 

Expert System applications are becoming more prevalent in fields where proper 
functioning is essential, such as the medical, financial, and aerospace industries. It is 
widely claimed that Expert Systems are not as rigorously evaluated as traditional 
software because of unique, unresolved evaluation issues. To ensure the continued 
and safe deployment of Expert Systems into critical areas, adequate evaluation tech- 
niques which address these issues must be developed and performed. 

The answers to this questionnaire, together with follow-up interviews, will provide 
realistic answers to the following questions: 

• How much evaluation is being performed? 

• What evaluation techniques are in use? 

• What, if any, are the unique issues in evaluating Expert Systems? 


Instructions 

The following questions concern your experiences with an Expert System, either as 
a user or as the manager of a department that uses Expert System. Feel free to 
indicate your answers in any way you like. Some of the choices on the multiple 
choice questions have places to fill in additional information; please indicate the 
choice and include the additional information, if possible. 

This survey task must be performed within a relatively short time period. If pos- 
sible, please return completed questionnaires within one week of receipt to: 

Keith Kelley 
MC 6272A 

IBM Federal Solutions Division 
3700 Bay Area Blvd. 

Houston, Tx. 77058-1199 

If you have any questions regarding this questionnaire, please contact Keith at (713) 
282-7303. Each participating project may request a copy of the final survey report 
from Keith. 


Questions 


1 . What is the name of the Expert System you were/are involved with? 


2. Are you a user of the Expert System or the manager of a department which 
uses the Expert System? 

a. User of the Expert System b. Manager of a department using 

the Expert System 

3. The responses that you provide in this questiqnnaire may indicate that further 
discussion is required for us to understand the issues that you encountered 
during the evaluation process. Would you be available, at your convenience, 
to discuss the evaluation process in more detail? Interviews will be fairly short 
one-on-one meetings either in person or by telephone. 

a. No, lam not available for an interview. 

b. Yes, I am available. 

Name 

Phone 


4. What field does the problem belong to? 


a. 

Aerospace 

g- 

Medical 

b. 

Financial 

h. 

Personnel 

c. 

Information Systems 

i. 

Research 

d. 

Hardware 

j- 

Service 

e. 

Manufacturing 

k. 

Software 

f. 

Marketing 

1 . 

Other 


5. Which of the following items best describes what the Expert System does? 
Please indicate primary purpose with a and check all other applicable pur- 
poses (if any). 

a. Design - Configuring objects under constraints 

b. Repair - Executing plans to administer prescribed remedies 

c. Control - Governing overall system behavior 

d. Planning - Designing actions 

e. Diagnosis - Inferring system malfunctions from observables 

f. Debugging - Prescribing remedies for malfunctions 

g. Prediction - Inferring likely consequences of given situations 

h. Monitoring - Comparing observations to expected outcomes 

i. Instruction - Diagnosing, debugging, and repairing behavior 

j. Interpretation - Inferring situation descriptions from sensor data 

k. Classification - Categorizing objects by properties 
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Some problems require the use of certainty factors (also called probabilities, or 
“fuzzy logic”) in their processing. Facts which contain certainty factors have 
the form: “if a is true, then there is an x% chance that b is true.” Does the 
Expert System include certainty factors? 

a. Yes c. I don't know 

b. No 

How much of the total problem space is the Expert System expected to 
address? I.e., if the Expert System is supposed to be able to diagnose 100 mal- 
functions, but the total number of malfunctions is known to be 200, the 
Expert System is expected to address 50% of the problem space. 


a. 

100% 

f. 

60% to 80% 

b. 

> 99% 

g- 

40% to 60% 

c. 

95% to 99% 

h. 

Other 

d. 

90% to 95% 

i. 

I don't know 

e. 

80% to 90% 




What is your estimate of the percentage of the problem space that the Expert 
System actually covers? 


a. 

Same as expected 

f. 

80% to 90% 


• 

b. 

100% 

g- 

60% to 80% 



c. 

> 99% 

h. 

40% to 60% 



d. 

95% to 99% 

i. 

Other 

_% 


e. 

90% to 95% 

j- 

I don't know 




When developing an Expert System, the person who provides the knowledge 
that is to be captured in the system is called the expert . For that part of the 
problem space addressed by the Expert System, how often is the expert(s) 
expected to give the correct answer? I.e., referring to question 7, how often 
does the expert(s) identify the correct malfunction out of the 100 addressed 
malfunctions? 


a. 

'Correct* defined by expert 

f. 

60% to 80% 



b. 

> 99% 

g- 

40% to 60% 



c. 

95% to 99% 

h. 

Other 

_% 


d. 

90% to 95% 

i. 

I don't know 


- 

e. 

80% to 90% 




* 

For that part of the problem space addressed by the Expert System, how often 

— 

is 

the Expert System expected to provide the correct answer? 



a. 

100% 

f. 

60% to 80% 



b. 

> 99% 

g- 

40% to 60% 



c. 

95% to 99% 

h. 

Other 

_% 


d. 

90% to 95% 

i. 

I don't know 



e. 

80% to 90% 
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11. 


What is your estimate of the percentage of the time that the Expert System 
provides the correct answer for that part of the problem space addressed by the 
Expert System? 



a. 

Same as expected 

f. 

80% to 90% 


b. 

100% 

g- 

60% to 80% 


c. 

> 99% 

h. 

40% to 60% 


d. 

95% to 99% 

i. 

Other % 


e. 

90% to 95% 

]• 

I don't know 

12. 

Was 

the expert(s) a member of the user organization? 


a. 

Yes 

c. 

User organization provided 


b. 

No 


some expertise 

13. 

Was 

the developer(s) of the Expert System part of the user organization? 


a. 

Yes 

c. 

User organization participated 


b. 

No 


in development 

14. 

Why do you believe the results that the system gives? 


a. 

Expert says it is correct 

e. 

User acceptance 


b. 

Participated in evaluation 

f. . 

I don't trust the results 


c. 

Someone I trust did evaluation 

g- ' 

Other 


d. 

Personal use and checking 



15. 

What is the worst thing that can happen if the 

Expert System gives the wrong 


answer? 




a. 

Someone gets hurt 

d. 

Workaround must be used 


b. 

Loss of "mission' 

e. 

Nothing 


c. 

Nuisance (correct answer 

f. 

Can't tell answer is wrong 



derived some other way) 

g- 

Other 

16. 

How does the number of errors that the users encounter compare with the 
number of errors they encounter with other systems which are not Expert 


Systems? 




a. 

Significantly more errors 

d. 

Fewer errors 


b. 

More errors 

e. 

Significantly fewer errors 


c. 

About the same number of 

f. 

No errors encountered 



errors 

g- 

I don't know 


If you were not involved with evaluating the Expert System, please leave the 
remaining questions unanswered. 
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17. What evaluation activities were performed on the executing system? (indicate 
any that apply) 


a. 

No evaluation was performed 

d. User acceptance 

b. 

Checked by expert(s) 

e. Other 

c. 

Compared with documented 



behavior 



18. During evaluation, the results from executing the system were compared with: 


a. 

Requirements document 

d. Majority opinion of experts 

b. 

System prototype 

e. Other 

c. 

Single expert 



19. What was the level of agreement among the experts concerning the correctness 
of the system? That is, do the experts agree on the correctness of the results of 
the system? Please note that this does not mean that the experts agree with 
the system, but rather, that they agree with each other about the results of the 
system. 

a. Always agree c. A single expert was involved 

b. Agree % of the time. 

20. How hard was the evaluation effort to perform? 


a. 

Trivial 

d. Hard 

b. 

Easy 

e. Impossible 

c. 

Medium 



21. How much effort was expended by the user group in evaluating the correctness 

of the Expert System? (please take a guess if you don't know) 

person/months. 

The indicated effort is: 

a. The actual effort c. A guess 

b. A very close approximation 
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